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EXECUTIVE  SUMMARY 

PURPOSE  AND  SCOPE 

The  work  presented  in  this  report  was  performed  under  Subtask  4  of  Task  Order  33  of 
the  SDIO  Systems  SETA  contract.  The  purpose  of  the  subtask  was  to  develop  a  plan  for  the 
introduction  of  software  measurement  into  the  SDS  development  program.  Sparta  was  the 
technical  lead  on  the  subtask,  with  support  from  Teledyne  Brown  and  TASC. 

The  subtask  was  broken  into  two  phases.  The  first  phase  focused  on  a  review  of  the 
results  of  Subtasks  1,  2,  and  3,  which  presented  SDS  Software  Measurement  requirement, 
models,  and  tools,  and  near  term  efforts  to  advance  SDS  Software  Measurement  plans.  The 
second  phase  focused  on  rhe  definition  of  activities  that  provide  for  a  stronger  foundation  for  the 
program  in  the  longer  term,  with  an  emphasis  on  model  development. 

REVIEW  OF  REQUIREMENTS,  METRICS,  TOOLS 

The  primary  purpose  of  software  metrics  is  to  predict,  throughout  the  software 
development  phase,  what  the  quality  and  overall  schedule  and  cost  of  the  final  product  will  be. 
This  prediction  process  is  to  be  integrated  with  the  standard  review  and  audit  activities  of  existing 
software  development  methodologies  for  the  SDS.  As  a  basis  for  quality  prediction,  nine  quality 
factors  were  defined.  Lines  of  code,  or  function  points,  were  the  primary  basis  for  productivity 
predictions.  The  SDS  software  was  decomposed  into  36  different  software  application  domains, 
in  order  to  produce  more  homogeneous  groupings  for  prediction  purposes.  A  framework 
methodology  requirement  was  defined,  consisting  of  specification,  estimation,  evaluation,  and 
tuning  phases. 

The  collection  of  software  metrics  available  for  use  proved,  as  was  expected,  to  be 
concentrated  late  in  the  software  development  life  cycle,  often  with  no  formal  foundations. 
Productivity  models  and  supporting  tools  are  more  mature  than  quality -oriented  models  and  tools; 
COCOMO  and  its  variations  are  in  widespread  use.  While  not  being  oriented  toward  prediction, 
the  software  quality  and  management  indicators  standardized  by  the  Army  and  Air  Force  can  assist 
in  providing  early  and  consistent  software  productivity  and  quality  statistics.  A  movement  toward 
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a  multi-attribute  vector  representation  of  quality  factors  is  identified.  Specific  tools  packages  which 
are  applicable  to  SDS  needs  are  "SMART"  (Software  Management  and  Reporting  Tool)  and 
"SMERFS"  (Statistical  Modeling  and  Estimation  of  Reliability  Functions  for  Software). 

NEAR  TERM  ACTIVITIES 

To  support  the  introduction  of  software  measurement  into  SDS  software  development,  a 
review  of  the  conclusions  and  proposals  presented  in  TR-9033-1  and  TR-9033-2  is  proposed.  A 
series  of  briefings,  with  audience  feedback,  is  defined  to  present  the  SDS  Software  Measurement 
materials  to  major  SDS  contractors  and  associated  Government  personnel.  The  intent  is  both  to 
educate  the  audience  on  SDIO  plans  in  this  area,  and  to  profit  from  their  evaluations  of  the  SDS 
Software  Measurement  approach,  based  on  their  unique  experiences  and  knowledge. 

Based  on  the  set  of  candidate  software  metrics  models  and  tools  previously  identified,  an 
initial  measurement  system  should  be  developed  for  SDS.  The  system  would  be  used  to  perform 
initial  experiments  in  software  characteristics  prediction,  with  a  small  number  of  software 
development  efforts.  Cost,  schedule,  and  reliability  predictions  would  the  the  focus  of  initial 
experiments  with  the  system. 

LONGER  TERM  ACTIVITIES 

Using  the  initial  SDS  Software  Measurement  System  for  experiments  to  gain  experience 
is  a  necessary  precursor  to  future  SDS  Software  Measurement  activities.  From  the  results  of  the 
experiments,  and  the  already  recognized  software  metrics  areas  of  deficiencies,  requirements 
would  be  identified  for  the  next  phase  of  development.  These  requirements  would  include  models 
for  reliability,  integrity,  portability,  usability,  and  reusability.  One  promising  area  of  development 
appears  to  be  in  multi- variate  vector  metrics,  as  it  has  been  successfully  applied  to  the  field  of 
general  systems  optimization  modelling. 
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In  addition,  the  measurement  methodology  itself  would  be  more  completely  defmed. 
This  definition  would  include  definition  of  SDS  software  development  methodology  to  include 
prototyping  methodology.  The  software  measurement  methodology  would  be  fully  integrated  with 
the  expanded  development  methodology.  The  redefinition  would  also  include  further  development 
of  the  software  development  methodology  to  incorporate  methodologies  for  rating  quality,  model 
tuning,  and  metrics  selection.  Finally,  an  integrated  system  of  automated  tools  would  be 
developed  to  support  the  entire  suite  of  SDS  Software  Measurement  functions. 
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0.  INTRODUCTION 

This  document  presents  the  tasks  to  be  accomplished  in  order  to  define  a  program  of 
software  measurement  for  SDS  development,  and  to  introduce  that  program  into  the  SDS  program. 
The  plan  focuses  on  both  near  term  and  longer  term  tasks  necessary  to  initiate  and  maintain  a  viable 
software  measurement  program  for  the  SDS.  Near  term  tasks  are  oriented  toward  review, 
assimilation,  and  refinement  of  a  proposed  SDS  Software  Measurement  approach,  and  use  of 
existing  software  metrics  tools.  Longer  term  tasks  target  modelling  of  software  development  life 
cycles  and  definition  of  additional  metrics  development,  validation,  and  automated  implementation. 

Section  1  discusses  the  importance  of  software  measurement  in  supporting  SDS  software 
development.  Section  2  reviews  the  findings  of  TR-9033-1  (SDS  Software  Measurement 
Requirements)  and  TR-9033-2  (Evaluation  of  Software  Measurement  Processes  and  Software 
Measurement  Support).  It  also  lists  near-term  activities  to  be  undertaken  to  move  the  SDS 
Software  Measurement  program  forward.  Section  3  presents  longer  term  activities,  including 
additional  software  metrics  validation  and  integration,  and  development  of  automated  tools  and 
databases.  Appendix  A  presents  a  multi-metric  measurement  model  with  a  formal  mathematical 
foundation. 
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1.  SDS  SOFTWARE  MEASUREMENT 

GOALS/OBJECTIVES 

Software  measurement  is  recognized  as  an  important  contributor  to  the  successful 
development  of  the  Strategic  Defense  System  (SDS).  While  the  exact  extent  of  the  software 
necessary  to  define,  simulate,  prototype,  implement,  test,  deploy,  and  maintain  the  SDS  is  not 
known,  a  common  indicator  of  the  scope  of  the  effort  has  been  the  estimate  of  between  10  and  90 
million  lines  of  code.  This  is  a  daunting  number,  even  to  those  who  previously  developed  large 
systems. 

Size  is  not  the  only  challenging  aspect  of  SDS  software  development.  Necessarily 
distributed  across  many  years,  it  also  will  likely  be  accomplished  by  several  dozens  of  individual 
contractors  and  subcontractors.  In  addition,  the  SDS  encompasses  a  broad  collection  of  differing 
functions,  ranging  from  mundane  (accounting,  formatting  tools,  etc  )  to  esoteric  (nuclear 
detonation  detection/analysis,  interceptor  guidance,  etc  ).  Some  needs  will  be  met  by  off-the-shelf 
software  products;  others  will  require  extended  development.  During  the  period  of  development, 
neither  development  technology,  system  design,  nor  hardware  base  can  be  expected  to  remain 
static.  New  technology  must  be  planned  for  and  accommodated  within  the  SDS  development 
plans. 


One  other  characteristic  of  the  SDS  has  a  major  impact  on  software  development.  SDS 
mission  critical  software  must  perform  as  designed.  The  consequences  of  SDS  software  failing  to 
perform,  or  performing  correctly,  but  under  the  incorrect  circumstances,  could  be  catastrophic. 
Thus,  the  criticality  of  the  mission  requires  the  most  reliable,  correct  software  w<*  can  build.  The 
costs  of  errors  in  that  software  are  grouped  in  two  categories,  then.  (1)  cost  of  failed  or  incorrect 
mission  accomplishment,  and  (2)  cost  of  preventing/identifying/removing  errors  in  the 
development  process.  Software  measurement  can  be  used  to  assist  in  minimizing  the  costs 
associated  with  (2)  above,  while  vigorous  testing  and  evaluation  addresses  overall  cost  of  error 
minimization. 

SDS  development  is  expected  to  be  phased  over  many  years,  with  hundreds  or  thousands 
of  interrelated  tasks  to  be  accomplished.  Software  development  must  be  accomplished  in 
accordance  with  the  overall  schedule.  Without  firm  monitoring  and  controls  on  progress,  software 
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development  has  the  potential  for  delaying  SDS  completion  by  perhaps  years.  In  addition,  the 
costs  associated  with  these  delays  could  escalate  to  prohibitive  levels. 

SDS  software  measurement,  then,  has  as  its  goal  to:  (a)  measure  the  product,  and 
process,  of  SDS  software  development,  both  to  predict  results,  and  to  allow  feedback  and  control, 
and  (b)  to  achieve  a  superior  product,  on  time  and  within  budget.  The  key  concept  here  is 
prediction.  With  accurate  prediction  early  in  the  software  development  life  cycle,  discrepancies 
between  expected  and  required  results  can  be  identified,  and  appropriate  steps  taken.  Without  the 
prediction  afforded  by  software  measurement,  problems  will  remain  hidden  longer,  requiring 
additional  resources  to  resolve.  Thus,  software  measurement  is  the  "early  warning  system"  in  the 
management  of  SDS  software  development. 
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2.  SDS  SOFTWARE  MEASUREMENT 

NEAR-TERM  APPROACH 

This  sectior  discusses  the  near-term  activities  to  be  undertaken  to  introduce  in  a 
systematic  manner  the  software  measurement  process  into  SDS  Software  Development.  It 
summarizes  the  findings  documented  in  TR-9033-1  and  TR-9033-2,  and  proposes  a  method  for 
review  of,  and  concurrence  with  those  findings,  by  Government  and  other  contractor  personnel. 
This  _ action  also  defines  an  incremental  approach  to  software  measurement  introduction,  using  an 
initial  tailored  software  metrics  program  with  one  or  more  development  contracts.  It  also  includes 
a  recommendation  for  experiments. 

2.1  SUMMARY  OF  TR-9033-1  AND  TR-9033-2 

This  section  summarizes  the  findings  from  Subtasks  1, 2,  and  3  concerning  measurement 
requirements  by  software  application  domains,  and  the  models,  metrics,  tools,  and  environments 
that  support  those  measurements. 

2.1.1  Summary  of  SDS  Software  Measurement  Requirements 
(TR-9033-1) 

The  work  presented  in  this  report  was  performed  under  Subtask  1  of  Task  Order  33  of 
the  SDIO  Systems  SETA  contract.  The  purpose  of  the  subtask  was  to  develop  SDS  software 
measurement  requirements  by  identifying  the  standards  and  attributes  required  for  SDS  software 
products  and  the  software  development  process. 

The  subtask  was  broken  into  three  phases.  The  first  phase  was  a  review  of  standards  and 
development  process  relevant  to  SDS,  as  they  pertained  to  software  metrics.  The  second  phase 
was  a  detailed  examination  of  SDS  software  characteristics,  definition  of  software  domains,  and 
quality  and  productivity  measurement  requirements.  The  third  phase  integrated  the  measurement 
requirements  identified  previously  into  an  approach  for  the  application  of  software  metrics. 
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Standards  and  Process  Identification  -  For  purposes  of  this  document,  software 
measurement  is  defined  as  the  act  of  capturing  metrics  and  comparing  them  to  standards.  A  metric 
is  defined  as  a  quantitative  standard  of  measurement  used  to  represent  and  compare  some  software 
process  or  product  attribute.  The  primary  objective  of  software  metrics  is  to  predict,  throughout 
the  development  phase  of  the  software,  what  the  quality  and  overall  schedule  and  cost  of  the  final 
product  will  be.  The  task  began  with  the  identification  of  potentially  relevant  standards  and 
development  guidelines.  Many  were  found  to  be  of  too  high  a  level  for  specific  guidance,  or  had 
no  relevance  to  software  metrics  technology.  The  ones  which  were  found  to  most  directly  address 
metric  requirements  were: 

a)  SDS  Software  Policy  and  Management  Directive  No.  7; 

b)  DoD-STD-2167A,  Defense  Systems  Software  Development; 

c)  RADC-TR-85-37,  Vols.  1,  2,  and  3,  Specification  of  the  Software 
Quality  Attributes. 

These  documents  formed  the  core  of  the  analysis,  as  they  most  directly  addressed  the 
development  process  and  the  associated  software  attributes.  Integration  of  software  metrics  into 
the  software  development  process  was  examined,  including  both  the  waterfall  development  method 
and  the  rapid  prototyping  development  method.  While  metrics  integration  with  the  waterfall 
development  method  was  relatively  straightforward  to  define  (see  Figure  2.1-1,  Software 
Acquisition  Quality  Metric  Functions),  integration  with  the  rapid  prototyping  development  method 
(itself  not  well  understood)  was  not  as  complete. 


Products  and  Process  Attributes  -  The  second  phase  of  this  subtask  was  to  define 
the  software  quality  and  productivity  factors  relevant  to  SDS  software.  This  identification  spanned 
the  software  development  life  cycle  from  requirement  definition  through  operation  and 
maintenance.  For  the  quality  factors,  the  thirteen  factors  identified  by  RADC  were  examined  and 
lyzed  for  applicability  to  SDS  needs.  Several  were  redefined  or  merged  together,  to  give  a  new 
set  of  nine  quality  factors.  Productivity  factors  were  also  examined,  with  factors  relating  to 
schedule,  budget,  and  feedback  for  improvement.  Primary  sources  for  analysis  were  based  on 
lines  of  code  estimates  for  software  components,  or  the  identification  of  function  points.  Both 
quality  and  productivity  metrics  are  believed  to  be  much  more  accurate  when  applied  against 
subsets  of  "like"  software  (software  domains).  Accordingly,  the  SDS  software  functions  were 
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MONITORING 


•  GATHER  DATA  AT  REVIEW  POINTS 


•  EVALUATE  DATA 


•  COMPARE  TO  REQUIREMENTS 


TRACK  PROGRESS 


•  CORRECT  DEFICIENCIES 
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Figure  2.1-1  Software  Acquisition  Quality  Metric  Functions 
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measurement  requirements  (Figure  2.1-2).  Each  domain  was  described  through  the  function 
implemented,  its  level  of  criticality,  time  constraints,  location,  size,  risk,  use,  and  intended  life 
cycle,  as  well  as  its  quality  attributes.  This  division  served  then  as  the  basis  for  further 
measurement  requirements  definition. 

Methodology  Requirements  -  The  third  phase  of  this  subtask  was  to  define 
methodology  requirements  for  incorporation  of  software  measurement  into  the  software 
development  methodology.  The  approach  defined  was:  specification,  estimation,  evaluation,  and 
tuning.  The  specification  phase  is  perhaps  the  most  difficult,  requiring  negotiation  among 
developers,  users,  and  contracting  agencies  to  set  specific  requirements.  The  estimation  phase 
uses  the  appropriate  metrics  to  obtain  predicted  measures  of  product  and  process.  Upon  delivery, 
the  user  evaluates  the  actual  product,  and  that  evaluation  is  compared  to  the  metrics  predictions. 
Finally,  the  metrics  are  adjusted  as  necessary  to  provide  more  accurate  assessments  for  future 
developments. 

Conclusions  -  There  were  several  major  conclusions  from  this  subtask.  The  results  of 
the  review  of  available  standards  revealed  that  formalization  of  the  process  of  estimating  quality  by 
embedding  it  in  the  development  process  is  not  well-defined.  Rapid  prototyping,  in  particular, 
must  be  directly  addressed.  For  the  waterfall  development,  the  incorporation  process  was 
identified.  A  modified  set  of  quality  factors  tailored  to  SDS  requirements  was  proposed,  reducing 
metric  application  requirements  while  targeting  specific  needs.  We  developed  a  methodology  for 
setting  quality  factors  based  on  SDS  software  characteristics.  The  SDS  software  was  decomposed 
into  36  software  application  domains  for  purposes  of  measurement  application.  The  concept  of 
software  level  to  determine  the  extent  of  software  metrics  application  was  described.  Several 
promising  productivity  metrics,  were  identified,  which  appear  applicable  to  the  SDS  software 
domains.  A  four  phase  methodology  requirement  of  software  metrics  application  was  also 
developed. 


' 

1 


2-4 


TASc! 

TIE 
SPARTA 
ARt  i 
6RC  * 
USA  1 

t  y*  trtwuTir  *r»  tiam  ii 

f  COMMTTTtT  TO  %DO  * 

THE  ANALYTIC  SCIENCES  CORPORATION 


0389/002*001 


Figure  2.1-2  SDS  Software  Application  Domains  and  Quality  Factors 
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Open  Issues  -  One  major  open  issue  is  the  establishment  of  procedures  for  audit  and 
review  activities  for  the  rapid  prototype  development  process.  After  that  is  complete,  then  software 
metrics  application  can  be  integrated  with  it.  A  formal  methodology  for  rating  quality  must  be 
developed.  Without  that,  unstable  metrics  models  can  result.  Also,  a  formal  methodology  to 
"tune"  metrics  models  must  be  defined.  This  includes  developing  formal  quality  rating  criteria,  as 
well  as  guidelines  for  modifying  scores.  Without  this  definition,  attempts  to  "trade  off’  one  quality 
factor  against  another  are  not  likely  to  provide  the  desired  results.  Finally,  software  reuse  must  be 
aggressively  encouraged.  Software  metrics  can  provide  estimates  of  software  reusability,  which 
can  be  compared  with  actual  reuse,  providing  a  measure  of  one  of  the  most  promising  sources  of 
cost/schedule  reduction. 

2.1.2  Summary  of  Recommended  Models  and  Metrics  for  SDS 
Software  Measurement  (TR-9033-2) 

The  collection  of  metrics  data  identified  in  Subtask  2,  TR-9033-2,  proved  to  be  both 
skewed  and  elusive.  Skewed  in  the  sense  that  much  of  the  existing  metrics  information  that  is 
found  to  exist  and  is  formalized  via  models  and  mathematical  relationships  is  encountered  late  in 
the  life  cycle  (i.e.,  occurring  in  the  implementation  phase  and  beyond).  Elusive  due  to  the  fact  that 
early  life  cycle  (predictive)  metrics  are  virtually  non-existent,  and  when  identified,  have  no  formal 
foundations  (mathematical  or  relational)  to  support  them  for  the  most  part.  Furthermore, 
consensus  is  still  evolving  on  the  valid  scopes  of  specific  metrics  applications  (e.g.,  complexity) . 

The  metrics  which  have  been  standardized  by  the  Air  Force  and  Army  for  use  throughout 
the  life  cycle  are  a  set  of  primitive  indicators  and  weighted  summations  for  quality  and 
management  control  and  accounting  rather  than  predictions.  Also,  productivity  models  have  been 
in  wide  use,  with  quite  a  rich  experience  base  as  to  fine-tuning  parameters;  the  second  generation; 
of  the  Constructive  Cost  Model  "COCOMO"  has  active  user  groups.  However,  one  message 
which  should  be  clear  out  of  this  study:  each  program  management  shop  should  establish  and 
implement  its  own  metrics  methods  and  quality  management  program  and  develop  its  own 
expertise  with  its  software  measurements  and  data. 

Though  there  has  been  no  early  life  cycle  predictive  model  identified,  the  use  of  the 
AFSCP-800-43,  -14  indicators  will  provide  early  and  consistent  software  productivity  and  quality 
statistics.  When  advanced  metrics  development  proceeds  newer  models  can  be  introduced  with  an 
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established  SDS  software  measurement  database  for  subsequent  validation.  Also  in  the  absence  of 
an  algorithmic,  predictive  model,  a  multi-attribute,  composite  or  vector  metric  representation 
presents  a  practical  approach  to  communicate  and  scale  an  assortment  of  primitive  indicators.  This 
use  of  multi -attribute  modelling  ("fuzzy"  and  simple)  has  been  recently  added  to  the  statistics  and 
O-R  literature  and  is  the  subject  of  the  1990  International  Conference  on  Metrics  being  held  in  the 
United  States  for  the  first  time  in  several  years. 

Software  Quality  and  Management  Indicators  -  The  review  of  the  metrics 
contained  in  Subtask  2  reveals  that  no  single  metric  is  capable  of  supporting  decisions  across  an 
entire  software  domain  as  defined  in  Subtask  1,  TR-9033-1.  Each  metric  supports  measurements 
in  one  or  two  software  attributes;  e.g.,  Reliability,  Maintainability,  etc.  The  implication  here,  is 
that  perhaps  several  metrics  will  need  to  be  combined  to  form  a  composite  model  capable  of 
supporting  decisions  on  the  36  SDS  subfunctions  or  the  defined  software  domains.  The  analysis 
concludes  that  no  available  metric  should  be  dropped  from  consideration. 

The  early  use  of  the  Air  Force  Standard  Quality  and  Management  Indicators  are 
recommended.  With  the  advent  of  the  Software  Management  and  Reporting  Tool  "SMART",  the 
collection  and  management  of  the  indicators  will  be  much  more  efficient  and  flexible.  Synopses  of 
the  Management  and  Quality  Indicators  are  presented  in  Table  2. 1.2- 1  and  2. 1.2-2. 

Composite  Metrics  and  Early  SDLC  Phase  Models  -  TR-9033-2  also  proposed 
the  creation  of  a  vector  which  could  be  used  to  assess  the  relative  strength  or  weakness  of  a  metric 
when  it  is  applied  to  one  of  the  36  SDS  subfunctions.  The  analysis  concluded  that  many  of  the 
relative  software  attributes  are  not  address  by  any  of  the  available  metric  models.  However,  it  was 
noted  that  current  research  is  being  conducted  at  several  universities,  in  industry  and  in 
government.  The  need  to  address  the  measurement  of  software  attributes,  early  in  the  software 
development  process,  is  well  known.  Several  approaches  were  evaluated  during  the  performance 
of  Subtask  2.  A  more  recent  approach  is  the  use  of  an  emerging  class  of  metrics  designated  as 
multi-attributes  or  multi-criteria  metrics.  Work  on  this  type  of  metric  is  currently  on-going  at 
universities,  some  as  close  as  George  Mason  University  in  Fairfax,  Virginia. 
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Table  2. 1.2-1  AFSCP  800-43  Software  Management  Indicators 


1.  Computer  Resource  Utilization  (CRU)  Indicator  : 

l.A  Metrics  Used: 

-  Planned  deliverable  resource  (Memory,  CPU,  I/O). 

-  Minimum  (Proposed  to  be  delivered)  deliverable  resource. 

-  Actual  utilization. 
l.B  Users: 

-  Development  contractors. 

-  Software  development  project  officers. 

-  Program  managers. 

-  Product  division, 
l  .C  SDLC: 

-  Requirements  definition  phase. 


2. 
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Software  Development  Manpower  Indicator  : 

2. A  Metrics  Used: 

-  Planned  and  actual  staff  deviation. 

-  Staff  losses. 

2.B  Users: 

-  Development  contractors. 

-  Software  development  project  officers. 

-  Program  managers. 

-  Product  division. 

2.C  SDLC: 

-  All  phases. 


3.  Requirements  Definition  &  Stability  Indicator : 

3. A  Metrics  Used: 

-  Total  number  of  software  requirements. 

-  Number  of  untraceable  s/w  requirements 

-  Number  of  untestable  s/w  requirements. 
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Table  2. 1.2-1  AFSCP  800-43  Software  Management  Indicators 

(Continued) 


3.B  Users: 

-  Development  contractors. 

-  Software  development  project  officers. 

-  Program  managers. 

-  Product  division. 

3. C  SDLC: 

-  Phases  :  requirements  definition,  design,  test  and  reviews  (i.e.  IIR,  SRR, 
SDR). 

4.  Software  Progress-Development  and  Test  Indicator: 

4.  A  Metrics  Used: 

For  the  development  progress  portion 

-  CSCI  design  completion 

=( units  100%  designedAotal  units  pers  CSCI)X100 

-  CSCI  unit  test  completion 

=( units  100%  coded  &  tested/total  units  per  CSCI)X100 

-  CSCI  integration  completion 

=( units  100%  integrated  into  a  CSCI/total  units  per  CSC)X100 
For  the  testing  progress  portion 

-  s/w  test  successfully  completed. 

-  Problem  reports  opened. 

-  Problem  reports  closed. 

4.B  Users: 

-  Development  contractors. 

-  Software  development  project  officers. 

-  Program  managers. 

-  Product  division. 

4.C  SDLC: 

-  Design,  code,  and  test  phases. 
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Table  2. 1.2-1  AFSCP  800-43  Software  Management  Indicators 

(Continued) 

5.  Cost/Schedule  Deviations  Indicator: 

5.A  Metrics  Used: 

-  Cost  variance=BCWP-ACWP. 

-  Schedule  variance=BCWP  -  BCWS. 

-  Cost  performance  Index(CPI)=BCWP/ACWP. 

-  Schedule  performance  index(SPI)=BCWP/BCWS. 

-  Estimate  at  compIetion(EAC) 

=ACWP  +  (BAC-BCWP)/(0.2SPI  +  0.8CPI). 

-  Variance  at  completion(VAC)=EAC  -  BAC. 

-  Percent  completed=BCWP  CUM/BAC. 

5.B  Users: 

-  Development  contractors. 

-  Software  development  project  officers. 

-  Program  managers. 

-  Product  division. 

5. C  SDLC: 

-  Reviews  at  all  phases. 

6.  Software  Development  Tools  Indicator : 

6.  A  Metrics  Used: 

-  Number  of  months  available  from  when  the  tools  are  required  to  when  the 
tools  are  expected  to  be  delivered. 

6.B  Users: 

-  Development  contractors. 

-  Software  engineers 

-  Software  development  project  officers. 

-  Program  chief  engineers 

-  Program  managers. 

-  Product  division. 
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Table  2. 1.2-1  AFSCP  800-43  Software  Management  Indicators 

(Continued) 

6.C  SDLC: 

-  Probably  during  design,  code,  and  test  phases. 
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Table  2. 1.2-2  AFSCP  800-14  Software  Quality  Indicators 

1.  Completeness  Indicator : 
l.A  Metrics  Used: 

-  Weighted  sum  of  components  defined  by  SDLC  milestone  delivery  schedule 
checklists 

1  .B  Users: 

-  Development  contractor’s  software  engineering,  test,  and  quality  organization. 

-  Software  development  project  officers. 

-  Program  managers. 

-  Product  division. 

1. C  SDLC: 

-  During  dem/val  SRR,  test  phases. 

2.  Design  Structure  : 

2.  A  Metrics  Used: 

-  Design  Structure^  w  D 

Where,  w  =  weight  associated  with  each  component  (0- 1 ). 

D  =  Components  defined  as  boolean  or  scored 
rating  of  structuring  policy  implementation. 

2.B  Users: 

-  Development  contractor's  software  engineering  and  test  organizations. 

-  Software  development  project  officers. 

-  Program  managers. 

-  Product  division. 

2.C  SDLC: 

-  FSD  (PDR,  CDR)  and  testing  phases. 
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Table  2. 1.2-2  AFSCP  800-14  Software  Quality  Indicators 

(Continued) 


3.  Defect  Density  Indicator  : 

3. A  Metrics  Used: 

-  (Cumulative  defects  encountered)/(total  number  of  units  per  CSCI). 

-  (Cumulative  defects  corrected )/( total  number  of  units  per  CSCI) 

3.B  Users: 

-  Development  contractor's  software  engineering,  test  and  quality  organizations. 

-  Software  development  project  officers. 

-  Program  managers. 

-  Product  division. 

3. C  SDLC: 

-  FSD  (PDR),  and  testing  phases. 

4.  Fault  Density  Indicator: 

4.  A  Metrics  Used: 

-  Cumulative  faults  (causes  of  failures)  /  total  number  of  units  per  CSCI. 

-  Cumulative  faults  corrected  /  total  number  of  units  per  CSCI. 

4.B  Users: 

-  Development  contractor's  software  engineering,  test,  and  quality  organizations. 

-  Software  development  project  officers. 

-  Program  managers. 

-  Personnel  performing  government  acceptance  of  the  CSCI. 

-  Product  division. 

4. C  SDLC 

-  FSD  (CDR),  acceptance  testing  phases. 

5.  Test  Coverage  Indicator  : 

5.  A  Metrics  Used: 

-  (No.  of  implemented  capabilities  tested)/(total  required  capabilities )X  100% 

-  (Software  structure  tested)/(total  software  structured  100% 
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Table  2.1. 2-2  AFSCP  800-14  Software  Quality  Indicators 

(Continued) 

5.B  Users: 

-  Development  contractor's  software  engineering,  test,  and  quality  organizations. 

-  Government  acceptance  personnel. 

-  Software  development  project  officers. 

-  Program  managers. 

-  Product  division 

5. C  SDLC: 

-  Functional  and  requirements  testing  phases. 

6.  Test  Sufficiency  Indicator  : 

6.  A  Metrics  Used: 

-  Remaining  faults(FR)=(PF  -  FP)X(UI/UT) 

-  Maximum  tolerance(MAXT)=c0  (FR) 

-  Minimum  tolerance(MINT)=c0  (FR) 

Where:  TF=number  of  faults  predicted. 

FP=number  of  faults  detected  before  s/w  integration  testing. 

UI=number  of  units  integrated. 

UT=number  of  units  in  the  CSCI. 

FD=number  of  faults  detected  to  date  during  test. 

Co.  c0  are  maximum  and  minimum  tolerance  coefficients 

6.B  Users: 

-  Development  contractor’s  software  engineering,  test,  and  quality  organizations. 

-  Government  product  acceptance  personnel. 

-  Software  development  project  officers. 

-  Program  managers. 

-  Product  division 
6.C  SDLC: 

-  s/w  integration  testing  phase. 
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Table  2. 1.2-2  AFSCP  800-14  Software  Quality  Indicators 

(Continued) 


7.  Documentation  Indicator : 

7. A  Metrics  Used: 

-  Documentation  Index(DI) 

Normalized  Summation  of  wl  x  D  +  w2  x  S 
where:  wl  and  w2  are  the  weights  associated  with  the  assessments  of  the  documentation 
and  source  listings  .  respectively. 

D  is  the  documentation  products. 

S  is  the  source  listing. 

7.B  Users: 

-  Development  contractor's  software  engineering,  logistics,  and  quality  organizations. 

-  Government  products  acceptance  personnel. 

-  Software  development  staff  officers. 

-  Logistics  staff  officers. 

-  Program  managers. 

-  Product  division. 

7.C  SDLC: 

-  s/w  products  acceptance  phase. 
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It  should  be  noted  that  the  result  obtained  in  calculating  a  multi-criteria  metric  (i.e. ,  a 
single  scalar  value)  provides  the  same  information  as  a  single  complexity  metric  value  -  of  limited 
use.  However,  it  is  felt  that  a  composite  vector  or  multi-attribute  vector  would  provide  greater 
insight  into  the  nature  of  the  life  cycle  activities  when  properly  instrumented  as  a  result  of  the 
visible  vector  components  or  coefficients  that  are  always  present  or  absent  in  some  cases.  Such  a 
requirements  maturity  or  stability  vector  would  be  of  the  form  R  =  Ai+Bj+Ck, 

where  the 

i  component  represents  a  traceable  requirements  activity  axis 
(e.g.,  user,  system,  performance,  derived) 

the  j  component  represents  a  document  maturity  component  axis 
(e.g.,  needs  document,  draft  document,  final  design  spec,  draft 
final  document) 

and  the  k  component  represents  a  design  review  component  axis 
with  related  life  cyc,!;  phase  information  (e.g.,  system  design, 
software  design,  test,  or  PDR/CDR/FDR  states). 

The  attenuation  or  amplification  of  the  vector  components  are  contained  in  the  matrix 
coefficients  for  i,  j,  k;  A,  B,  C  respectively  of  form: 

[an  ai2  ...  ajn] 

[a21  a22  ...  a2n] 

[aml  am2  •••  amnl 

Requirements  synthesis  vectors  in  the  early  life  cycle  development  phases  prior  to 
developer  source  selection  activities  would  be  of  the  form: 

R  =  Ai+Bj 
or 

R  =  Ai 

A  vector  result  of  one  of  the  latter  two  types  in  the  development  phase,  after  contractor 
selection,  would  be  of  immediate  suspect  and  concern  to  a  program  manager  since  the  resultant 
vector  should  be  of  the  form  Ai+Bj+Ck. 
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It  would  be  obvious  that  a  development  component  is  missing,  possibly  as  a  result  of 
major  problems  or  exceedingly  high  resource  consumption  rates. 

This  is  one  approach  that  appears  to  shed  greater  information  on  requirements  stability, 
than  multi-criteria  metrics  and  is  in  its  early  stages  of  development.  Vector  metrics  can  be  plotted, 
with  associated  derivatives,  to  assess  work  progress  or  instability  points  along  a  critical  path. 
Constant  cost  curves  or  surfaces  can  be  generated  by  examining  vector  magnitudes  to  assist  in  the 
trade-offs  of  resource  consumption  and  risk.  It  is  felt  that  a  finer  granularity  can  be  achieved  over 
existing  cost  models  with  the  introduction  of  vector  metrics. 

Vector  metrics  implementation  is  based  on  the  assumption  that  predictive  measures  and 
measurands  can  be  identified  instrumented  and  automated  via  a  requirements  tool  set.  One  such 
tool  set  can  be  identified  establishing  several  possible  implementations  and  a  potential  series  of 
experiments  is  with  the  use  of  the  RVTS  tool  suite  and  the  predictive  measurands  identified  in 
Table  2. 1.2-1. 

2.1.3  Summary  of  Recommended  Tools  for  SDS  Software 
Measurements 

The  following  subparagraphs  summarize  the  findings  from  Subtask  3  concerning  the 
tools  and  environments  that  support  the  selected  measurement  processes  and  metrics. 

Software  Management  and  Report  Tool  (SMART)  -  The  SMART  package  is 
targeted  for  deployment  at  the  U.S.  Army  Communications  -  Electronics  Command  (CECOM) 
July-August  1989.  The  first  program  it  will  be  applied  on  is  the  Advanced  Field  Artillery  Tactical 
Data  System  (AFATDS). 

The  software  metrics  goad  is  to  implement  the  Software  Management  and  Quality 
Indicators  as  per  AFSCP-800-43  and  AFSCP-800-14  in  an  efficient,  extensible  architecture.  The 
software  metrics  indicators  and  data  management  will  be  based  on  IBM  PC-compatible  machines 
using  the  popular  "dBase3"  formats  with  the  "Clipper"  data  base  language  system. 

The  list  below  shows  some  of  typical  project  control  data  used  by  SMART  to  collect  the 
software  management  and  quality  indicators. 
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OVERVIEW  OF  SMART  SOFTWARE  DATA  COLLECTIONS 

Deviations 
Waivers 

Software  Trouble  Reports 
Test  Incident  Forms 
Engineering  Change  Proposals 
Software  Improvement  Reports 
Software  Discrepancy  Reports 
Computer  Resource  Utilization 
Software  Development  Manpower 
Requirements  Definition  and  Stability 
Software  Progress  -  Development  and  Test 
Cost/Schedule  Deviations 
Software  Development  Tools 
Completeness 
Design  Structure 
Defect  Density 
Fault  Density 
Test  Coverage 
Software  Maturity 
Documentation 

Statistical  Modelling  and  Estimation  of  Reliability  Functions  for  Software 
(SMERFS)  -  The  Farr  and  Smith's  SMERFS  package  of  reliability  models  from  the  Naval 
Surface  Warfare  Center  (NSWC)  at  Dahlgren  presents  a  field  tested,  efficient  tool  for  the  SDS  to 
use  on  an  interim  basis.  As  the  interim  use  of  the  SMERFS  models  progresses,  concurrent 
validations  and  research  into  highly  reliable  system  requirement  issues  can  be  used  to  tune  the 
models'  deployment  and  to  plan  for  an  advanced  model.  The  aspect  of  applying  uncertainty  theory 
should  also  be  examined  during  this  interim  period,  so  that  the  potentiality  of  applied  uncertainty 
modelling  can  be  assessed. 
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SMERFS  was  developed  several  years  ago  as  an  aid  in  the  evaluation  of  software 
reliability.  In  its  original  design  it  was  targeted  for  mainframe  and  mini-computer  environments. 
Since  then  it  has  also  been  adapted  to  operate  on  micro-computers,  specifically  IBM-PC/XT 
compatibles. 

The  current  version  of  SMERFS  has  incorporated  eight  software  reliability  models.  The 
models  include  the  following:  (1)  Musa's  Execution  Model,  (2)  Goel-Okumoto  Non- 
Homogeneous  Poisson  Model,  (3)  Adapted  Goel-Okumoto  Non-Homogeneous  Poisson  Model, 
(4)  Moranda's  Geometric  Model,  (5)  Schafer's  Generalized  Poisson  Model,  (6)  Schneidewind's 
Model,  (7)  Littlewood-Verrall  Bayesian  Model,  and  (8)  the  Brooks-Motley  Model. 

SMERFS  contains  a  driver  which  is  claimed  to  make  it  machine  independent.  The  driver 
is  a  subset  of  the  American  Standards  Institute  (ANSI)  specifications  for  the  FORTRAN  77 
compiler.  Several  user  selectable  options  are  available  within  the  driver  and  allow  the  system  to  be 
configured  to  produce:  better  predictions;  output  plots  and  catalogued  output  files.  Currently 
SMERFS  is  operational  on  three  main  computer  groups  at  the  Naval  Surface  Weapons  Center 
(NSWC),  Dahlgren,  VA.  The  three  computer  groups  include  the  CDC  CYBER  170/875,  the 
Vaxcluster  1 1/785,  and  a  large  number  of  IBM-compatible  PCs.  Dr.  William  H.  Farr,  of  NSWC, 
and  Mr.  Oliver  D.  Smith,  of  EG&G  Washington  Analytical  Services  Center,  Inc.  both  claim  that 
transferring  SMERFS  to  other  computers  should  be  very  easily  accomplished. 

Besides  containing  operating  instructions  within  its  interactive  mode,  SMERFS  two 
additional  pieces  of  documentation.  The  two  supplemental  reports  are:  (1)  SMERFS  Library 
Access  Guide  (NSWC-TR-84-371,  Rev.  1),  and  (2)  SMERFS  User's  Guide  (NSWC-TR-84-373, 
Rev.  1).  These  two  publications  allow  a  potential  user  to  preview  the  system.  Examples  are 
provided  throughout  the  User's  Guide,  allowing  a  potential  user  to  acquire  an  overview  of  the 
SMERFS  processing.  In  addition,  the  guide  also  shows  actual  software  reliability  analyses 
performed  on  the  CDC  CYBER  170/875. 

The  SMERFS  systems  show  tremendous  potential  for  use  in  the  SDI  environment  with  a 
minimum  of  modification. 
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ADC  Packages  AMS,  QES,  ASQS  -  These  three  programs  at  the  Rome  Air 
Development  Center  are  in  different  life  cycle  phases  as  described. 

Automated  Measurement  System  (AMS)  -  The  Automated  Measurement  System 
(AMS)  was  developed  by  the  Harris  Corporation  for  the  Rome  Air  Development  Center  to 
implement  the  RADC  Software  Quality  Metrics  as  specified  in  RADC-TR-85-37.  The  RADC- 
Harris  goal  was  to  interface  with  prevalent  coding  languages  (Fortran,  Ada,  COBOL)  and 
specification  languages  (SREM/RSL,  SDDL)  on  a  common  computer  system  platform  (DEC 
VAX,  VMS,  CMS),  written  in  Fortran,  with  off-the-shelf  (OTS)  software  reused  from  prior 
RADC  and  contractor  activities. 

The  development  was  generally  successful  with  a  few  significant  shortfalls.  Particularly, 
the  following  partial  language  interpretations  are  reported  as  disappointments: 

Partial  Language  Support  Percentages 


Ada 

36% 

Fortran 

60% 

SDDL 

46% 

SREM/RSL 

10% 

Verbal  reports  indicate  that  the  contractors  performance  testing  in  regard  to  the  Ada 
interface  has  not  been  satisfactorily  duplicated.  The  final  report  indicates  an  unsatisfactory  porting 
to  an  IBM  PC/AT  of  the  AMS.  Oral  reports  indicate  that  the  attempt  was  totally  unsuccessful.  The 
use  of  AMS  by  the  SDIO  for  Fortran  and  some  Ada  may  be  helpful  in  a  VAX  environment. 

Quality  Evaluation  System  (QES)  -  The  RADC  program  titled  Quality  Evaluation 
System  (QES)  is  also  called  "son  of  AMS"  by  some  RADC  staff.  This  package  will  be  written  in 
Ada  to  support  both  Ada  and  Fortran  applications.  The  target  date  is  late  1989  or  early  1990. 
Progress  of  this  package  should  be  tracked  by  the  SDIO. 


Automated  Specification  Quality  System  (ASQS)  -  The  RADC  has  begun  work 
on  an  Auiomated  Specification  Quality  System  (ASQS)  for  early  SDLC  phase  reporting. 
Development  of  this  program  should  be  tracked  by  the  SDIO. 
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2.2  REVIEW  OF  FINDINGS  AND  RECOMMENDATIONS 

The  SDS  software  measurement  requirements  documented  in  TR-9033-1,  and 
summarized  in  2.1.1,  were  developed  with  an  eye  toward  refinement  and  expansions.  Similarly, 
while  a  vast  array  of  information  was  amassed  and  evaluated  in  completing  TR-9033-2,  the  task 
was  necessarily  incomplete,  given  the  resources  available  and  the  constantly  changing  set  of  new 
products  being  developed.  Therefore,  a  critical  review  of  these  efforts  by  other  knowledgeable 
professionals  is  considered  not  only  prudent,  but  necessary,  to  achieve  a  consensus  on  the  SDS 
Software  Measurement  approach.  Another  purpose  in  having  review  of  SDS  software 
measurement  plans  is  to  more  closely  involve  affected  participants  in  the  process.  Besides  having 
expert  expe:'ence  in  diverse  areas,  many  of  the  reviewers  will  be  involved  in  initiating  the 
measurement  program  and  in  operating  within  the  requirements  of  that  program.  Thus,  they  will 
have  the  opportunity  not  only  to  learn  abort  requirements  that  may  be  imposed  upon  them,  but  also 
to  shape  those  requirements.  This  can  be  a  key  factor  in  the  early  establishment  and  success  of  the 
program. 

The  review  process  has  the  following  steps: 

a)  Identify  target  review  audiences 

b)  Group  attendees  for  briefings  as  desired 

c)  Prepare  briefing  material 

d)  Distribute  review  material 

e)  Present  briefings  and  receive  immediate  feedback 

f)  Analyze,  assimilate  feedback 

g)  Modify  SDS  Software  Measurement  approach  as  necessary 

h)  Prepare  and  distribute  specific  standards,  policies,  and  plans  for 
implementation. 

The  conclusions  contained  in  TR-9033-1  and  TR-9033-2  should  be  distributed  to 
"interested  parties”  in  the  field  of  SDS  software  development.  They  would  include:  a)  the  SE&I 
contractor;  b)  Software  Center  personnel;  c)  SPO  personnel;  0  procurement  personnel;  g) 
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software  tools  and  environment  developers;  h)  software  measurement  academicians;  and  i)  others 
as  deemed  appropriate  by  SDIO. 

While  not  all  the  reviewers  would  be  interested  in  all  aspects  of  the  reports,  the  analysis 
and  reasoning  in  the  technical  reports  would  be  useful  in  providing  a  good  background  for  review. 
For  example,  a  software  measurement  academician  would  primarily  be  interested  in  the 
measurement  approach  and  metrics  models;  however,  the  identification  of  the  36  SDS  software 
application  domains  would  be  useful  information  to  know,  as  domain  definition  is  an  integral  part 
of  our  approach. 

The  reviewers  should  concentrate  on  the  following  areas,  with  emphasis  on  areas  that 
directly  affect  them: 

a)  The  proposed  software  typing  scheme  and  resultant  candidate  generic 
software  domains; 

b)  The  relative  importance  of  the  software  quality  factors  within  each 
domain; 

c)  The  proposed  baseline  methodology  for  setting  specific  goals  or  targets 
for  software  quality  factors; 

d)  The  metrics  models  and  tools  designated  as  appropriate  for  each  domain 
and  factor  or  attribute;  and 

e)  The  proposed  application  and  feedback  processes  for  measurement  of 
the  SDS  software. 

The  reviewers  could  potentially  contribute  substantially  to  the  successful  definition  and 
use  of  software  measurement  in  SDS,  given  that  they  may  have  a  narrower,  but  deeper,  experience 
base  (e.g.,  the  BSTS  contractor,  in  BSTS  software  functions).  Also,  with  a  good  understanding 
of  the  measurement  program,  more  volunteers  for  pilot  experiments  would  be  likely. 

The  material  to  be  reviewed  (either  existing  documents  or  abstracts)  should  be  distributed 
to  the  reviewers  in  the  near  future.  Concurrent  with  the  distribution,  SDS  Software  Measurement 
briefings  should  be  developed  and  given  to  the  reviewers.  This  approach  allows  tailoring  of  the 
material  to  a  specific  audience.  For  example,  briefings  to  academicians  might  concentrate  on  the 
models  chosen,  while  briefings  to  major  component  contractors  would  emphasize  the  development 


2-22 


THE  ANALYTIC  SCIENCES  CORPORATION 


of  the  SDS  software  application  domains.  Tailored  briefings  also  allow  for  receiving  immediate 
verbal  feedback.  Without  the  briefings,  written  feedback,  possibly  weeks  or  months  later,  could 
be  the  only  response. 

Alternatively,  if  the  number  of  briefings  required  to  address  each  review  group  is  deemed 
excessive,  briefings  could  be  combined  to  address  several,  or  perhaps  all,  reviewers  at  one  time. 
A  simple  briefing  session  (2-3  days)  would  require  considerably  more  organization  time,  and 
probably  lead  time  as  well,  but  would  have  the  advantage  of  each  review  group  hearing  first  hand 
the  comments  and  ideas  of  the  other  reviewers.  In  addition,  while  set  up  time  would  be  increased, 
the  overall  time  to  distribute  material,  present  the  briefings,  and  receive  feedback  could  be  less  than 
that  necessary  if  multiple,  targeted  briefings  were  given.  As  noted  above,  one  of  the  key  review 
areas  is  the  set  of  tools  and  software  metrics  appropriate  for  a  project.  To  better  acquaint  the 
audience  with  the  process,  a  step  by  step  presentation  of  setting  the  software  measurement  goals 
for  a  project,  and  determining  the  measurements  to  be  taken,  should  also  be  given.  This  is  a 
particularly  recommended  approach. 

Following  the  briefing(s)  and  review  comments  assimilation,  the  comments  should  be 
reviewed  for  possible  incorporation  into  the  SDS  Software  Measurement  program.  Software 
domains  should  be  refined  in  light  of  the  then-current  understanding  of  the  systems  functioi  s. 
Quality  factor  rankings  should  be  adjusted  as  necessary,  and  the  software  measurement  toolset 
specified.  The  modified  standards,  policies,  and  plans  should  receive  wide  distribution,  and 
implementation  should  begin.  We  recognize  that  this  is  not  a  one-time  effort;  it  will  continue 
throughout  the  development  effort.  However,  it  is  necessary  to  assemble  the  program  with  best 
and  most  complete  information  available  at  that  time,  and  get  it  started.  Otherwise,  a  fragmented 
and  partial  approach,  at  best,  is  the  likely  alternative. 

2.3  DEFINITION  AND  DEVELOPMENT  OF  MEASUREMENT  SYSTEM 

Just  as  it  is  very  important  to  get  the  SDS  Software  Measurement  Program  off  to  an  early 
start,  so  too  should  the  Measurement  System  that  implements  the  models  and  software  metrics  of 
the  program  be  initiated  immediately.  Initially,  the  Measurement  System  will  be  less  a  complete 
system  than  a  collection  of  selected  tools  as  identified  in  Section  2. 1 .  The  emphasis  is  on  off-the- 
shelf  availability,  with  the  more  mature  tools  given  preference  over  beta-release  products.  Other 
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than  perhaps  some  tailoring  of  the  tools  to  the  SDS  Software  Development  environment,  no 
extensive  development  effort  should  be  initially  undertaken.  Experience  must  be  gained  with  the 
initial  toolset  before  major  additional  system  enhancements  are  initiated. 

Enhancements  will  be  identified  through  two  parallel  paths.  First,  the  gaps  identified  in 
TR-9033-2  for  tools  support  of  quality  measurements  must  be  filled.  Those  cases  where  a  metric 
is  identified,  but  no  tools  implements  that  metric,  should  be  addressed  first.  Here,  the 
requirements  are  more  completely  specified,  and  near  term  system  improvements  are  more  readily 
achievable.  For  those  cases  where  a  measurement  requirement  has  been  identified,  but  no 
approach  has  been  identified  or  found  to  be  acceptable,  additional  research  will  be  necessary  (see 
Sections  3.1  and  3.2).  Priority  for  development  of  software  metrics  tools  should  be  given  to  those 
metrics  which  are  associated  with  quality  factors  ranked  "High",  "Medium",  and  "Low",  in  that 
order. 

The  other  source  of  system  enhancement  requirements  is  that  resulting  from  use  of  the 
measurement  system.  Following  its  initial  definition,  users  of  the  system  will  identify  areas  of 
improvement.  Some  expected  areas  would  be:  a)  user  interface;  b)  integration  of  tools;  and  c) 
performance.  Integration  of  the  toolset  and  performance  would  be  particularly  critical  areas  for 
improvement.  If  the  tools  comprising  the  Measurement  System  are  not  well  integrated,  require 
duplicative  data  gathering,  or  wok  at  cross  purposes,  resources  will  be  wasted.  Of  paramount 
importance,  if  the  tools  themselves  do  not  produce  the  results  expected,  they  must  be  revised  and 
improved  as  soon  as  possible.  Otherwise,  the  Measurement  System’s  value  would  be 
questionable. 

As  noted  in  TR-9033-2,  most  metric  tools  available  today  deal  with  the  later  parts  of  the 
development  life  cycle  (i.e.,  code).  Given  the  emphasis  on  the  value  of  predictions  of  SDS 
software  quality  and  process  factors,  automated  support  for  evaluating  activities  earlier  in  the  life 
cycle  is  essential.  Thus,  while  the  initial  focus  of  the  Measurement  System  will  be  on  the  code 
itself,  the  early  enhancements  should  also  concentrate  those  tools  that  aid  in  early  evaluation, 
controlling,  and  predicting  of  software  characteristics.  These  early  qualitative  enhancements 
(spanning  more  of  the  development  process)  may  well  be  more  valuable  than  quantitative 
enhancements  (a  deeper  or  different  view  of  a  particular  measurement). 
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2.4  SELECTION  OF  PILOT  CONTRACTS 

Given  the  embryonic  nature  of  the  SDS  Software  Measurement  Program,  it  should  not  be 
imposed  across  all  on-going  and  future  software  development  efforts  without  initial  tryouts.  It  is 
important  to  gain  experience  with  the  program  under  controlled  circumstances  prior  to  full-scale 
introductions.  Volunteers  should  be  solicited  from  among  those  participating  in  the  Software 
Measurement  Program  review.  To  gain  initial  experience,  a  few  (two  or  three)  software 
development  efforts  should  be  picked  for  introduction  of  the  Software  Measurement  Program. 
These  efforts  should  be:  a)  relatively  small;  b)  well-defined;  and  c)  of  short  duration.  Efforts 
with  these  characteristics  should  be  picked  to  allow  for  rapid  assessment  of  results,  and 
minimization  of  outside  influences. 

As  noted  in  TR-9033-1,  the  software  measurement  process  should  be  tailored  to  each 
specific  effort.  Initially,  the  software  quality  specifications  should  be  defined.  This  is  a 
cooperative  effort  among  the  developer,  users,  the  System  Program  Office,  and  quality  assurance 
personnel.  Out  of  this  effort  is  accomplished:  a)  software  criticality  definition;  b)  technology  risk 
assessment;  and  c)  software  quality  criteria  and  targets  specification.  The  software  product  must 
be  assigned  to  the  appropriate  software  domain,  and  the  associated  quality  and  productivity 
requirements  defined,  and  refined,  as  appropriate. 

Following  the  definition  and  ranking  of  the  quality  factors,  the  appropriate  software 
metrics  models  and  tools  would  be  selected  from  the  available  set  for  application  throughout  the 
development  process.  The  contractor  then  would  initiate  development,  integrating  the  selected 
metrics  tools  and  procedures  with  the  development  process.  The  reports  and  analyzes  output  from 
the  measurement  system  would  then  be  reported  on  at  the  appropriate  points  in  the  review  process, 
allowing  for  continuing  assessment  of  the  software.  As  results  become  available,  they  could  be 
compared  with  the  objectives  set  during  development  initiation.  Deviations  from  objectives  would 
be  noted,  explained,  and  appropriate  action  taken  to  adjust  either  the  objectives,  the  development 
process,  or  both.  This  process  of  continuing  monitoring,  assessment,  and  adjustment  would 
continue  through  the  life  cycle  until  product  delivery.  At  that  time,  direct  user  assessment  of  the 
developed  system  would  be  compared  to  the  predictors  which  were  developed  from  application  of 
the  software  development  process.  From  this  comparison,  the  value  of  the  software  measurement 
process  in  providing  accurate  predictors  of  software  attributes  would  be  determined. 
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It  is  from  this  initial  application  of  the  Software  Measurement  Program  in  the  "real- 
world"  that  will  give  SDIO  the  feedback  necessary  to  determine  the  future  course  of  the 
measurement  effort.  If  the  results  reasonably  meet  expectations,  then  additional  contracts  should 
be  selected  for  use,  and  the  effort  expanded.  If,  however,  the  results  are  not  particularly  good, 
then  either  modifications  to  the  Software  Measurement  Program  must  be  made,  and  initial  trials 
re-executed,  or  the  utility  of  the  entire  effort  may  be  questioned. 

In  order  to  provide  a  high  degree  of  control  and  consistency  of  results  in  the  initial  trials, 
it  would  be  advisable  to  have  the  Software  Center  administer  the  Measurement  Program.  As  the 
probable  repository  of  the  complete  set  of  tools  and  environments,  they  would  have  the  expertise 
and  resources  necessary  to  support  the  software  measurement  program’s  introduction. 

2.5  EXPERIMENTATION  AND  FEEDBACK 

Specific  experiments  to  be  undertaken  include  primarily  those  which  address 
measurement  of  the  software  development  process.  Prediction  of  size,  cost,  and  schedule  are  most 
readily  undertaken,  and  would  likely  produce  the  most  unambiguous  results.  Experiments  with  a 
COCOMO-based  system  and  SOFTCOST-Ada  are  recommended  for  schedule  and  cost. 

For  experiments  with  software  quality,  software  reliability  is  the  factor  is  best  covered  by 
existing  metrics.  Experiments  with  SMERFS  are  recommended  to  get  reliability  predictions. 

The  above  experiments  should  either  be  performed  by  the  Software  Center  or  by  the 
contractors  selected  for  initial  Software  Measurement  trials. 
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3.  SDS  SOFTWARE  MEASUREMENT 

FUTURE  APPROACH 

This  section  discusses  activities  which  should  be  undertaken  in  the  longer  term  to  provide 
for  continued  improvement  in  the  SDS  Software  Measurement  process.  Key  among  these  are: 

a)  continued  development  of  models  and  metrics 

b)  data  collection  and  experimentation  to  validate  models  and  metrics 

c)  development  of  comprehensive  measurement  methodology 

d)  integration  of  measurement  with  the  SDS  software  development  process 

e)  definition  and  development  of  automated  tools  and  databases. 

3.1  SOFTWARE  MODELS  AND  METRICS  DEVELOPMENT 


There  should  be  a  two  pronged  evolutionary  approach  to  develop  the  SDS  Software 
Models  and  Metrics  consisting  of  the  following  activity  groups  of  equal  value: 

a.  Applying  currently  available  models. 

b.  Software  Metrics  Requirements  R&D. 

3.1.1  Evolutionary  Development 


An  evolutionary  approach  is  typically  used  when  the  knowledge  base  must  mature  to 
achieve  both  long-range  and  intermediate  objectives.  These  objectives  are: 


a.  To  develop  the  SDS  software  metrics  management  and  validation  data 
through  state-of-the-art  experience  with  off-the-shelf  (OTS)  package 
applications. 

b.  To  feed-forward  management  evaluations  and  validations  into  the  future. 


c.  To  develop  advanced  SDS  reliability  models  including  factors  for 
subprototype  replacement,  Ada  packaging  architecture  and  functional 
criticality  and  tolerance. 


d.  To  develop  appropriate  models  for  SDS  Integrity,  Portability,  Usability 
and  Reusability. 
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e.  To  develop  composite  metrics  with  meaningful  weightings  and/or  vector 
presentation. 

These  objectives  are  approached  by  means  of  short-range  and  long-range  activity  groups 
which  must  interrelate.  The  short-range  activities  must  feed-forward  experience  and  evaluations. 
The  long-range  activities  must  provide  refinements  and  validations  to  the  day-to-day  activities  to 
promote  accuracy  and  effectiveness. 

Figure  3.1-1  shows  an  overview  of  the  activity  groups. 

3.1.2  Short-Range  Activities 

The  short  range  activities  are  needed  to  provide  control  and  a  basis  for  validations.  As 
discussed  in  TR-9033-1  and  its  related  briefing,  the  application  of  the  Air  Force  Quality  and 
Management  Indicators  appear  to  be  a  simplistic  approach  to  tracking  quality  and  productivity,  but 
will  provide  a  controlled  footing  for  the  program.  Soon  with  the  advent  of  the  Software 
Management  and  Reporting  Tool  "SMART",  the  deployment  and  use  of  the  indicators  can  be 
efficiently  managed. 

Figure  3.1-2  summarizes  the  short-range  activities. 

Software  Indicators  for  Preliminary  Data  -  The  early  use  of  the  software 
management  and  quality  indicators  will  provide  a  beginning  for  the  SDS  software  database.  The 
efficient  application  of  the  Air  Force  pamphlets,  with  contractual/SOW  enforcement  for  the  data 
collections  will  start  to  build  the  productivity  and  quality  control  data  needed  for  early  management 
control.  More  importantly,  this  early  step  will  provide  a  basis  for  the  preliminary  validations  of 
assessment  models. 

As  experience  builds,  adjustments  to  SOWs  should  be  planned  for  additional  and  refined 
data  collections. 
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Feed  forward  experience  data 


Incremental  R&D  results 


Figure  3.1-1  Short-  and  Long-Range  Activities 
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Figure  3.1-2  Short-Range  Activities 


Build  SDS  Software  Database  -  As  soon  as  practical,  the  development  of  an  SDS 
software  database  should  be  started.  A  good  starting  basis  is  the  data  tracked  by  the  indicators 
package  Software  Management  and  Reporting  Tool  "SMART".  A  preliminary  checklist  of 
software  data  elements  from  the  early  phases  of  the  SDLC  is  presented  in  Table  3.1-1. 
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Table  3.1-1  Preliminary  Requirements  and  Early  SDLC  Measurands 

1 .  Requirements  Maturity/Requirements  Specificity 

2.  Level  (e.g.,  System,  Subsystem) 

3 .  Stability /Arbitrariness 

4.  Increment  Tools  Existence 

5 .  Functional  Coupling  Rating  (Functions  Mapped  to  Requirements) 

6 .  Derived  Requirements  Coupling 

7 .  Requirements  Traceability  Matrix  (yes  or  no) 

8 .  Traceability  Rating  (scored  or  descriptive) 

9.  Requirements  Tools  Deployment 

10.  PTRs/DTRs  Opened/Closed  by  Period 

1 1 .  User  Qualification/Validation  Ratings 

12.  Ratio  of  Validated  Requirements  to  TBDs,  Unknowns  and 
Preliminaries 

1 3 .  Prototype  or  Production 

14.  IV& V  Verification  Ratings  of  Functional  Definitions 

1 5 .  Development,  QA  &  IV& V  Staffing  Actual/Budget  Ratios 

1 6.  System  Spec.  Maturity  Score 

1 7 .  ROC/SOW  Mapping  Scores 

18.  Percentage  Replacement/New  &  Reuse/Unique 

19.  Development  Environment  Completion  Rating 

20.  Conflicting  Requirements  Weighted  Scoring 

2 1 .  Support  Requirements  Maturity  &  Completion 

22.  Requirements  Change  Drivers 
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Preliminary  Predictions  -  In  parallel  with  the  deployment  of  the  software  quality  and 
management  indicators  and  database  development,  the  program  should  start  the  preliminary  use  of 
available  models.  The  early  use  of  the  Statistical  Modeling  of  Reliability  Functions  for  Software 
(SMERFS)  eight  models  with  SDS  data  and  with  borrowed  data  from  similar  projects  will  provide 
the  type  of  practical  experience  needed  for  later  refinements  and  validations.  Also,  the  use  of  the 
SECOMO  and  REVTC  project  schedule  and  cost  modeling  packages  will  provide  management  with 
pilot  experience  needed  to  develop  an  efficient  management  and  control  organization. 

Both  the  cost/schedule  data  with  COCOMO/SECOMO/REVIC  and  the  reliability  data 
with  the  reliability  models  should  be  organized  and  preserved  for  subsequent  validations  and  model 
refinements. 

Preliminary  Validations  and  Adjustments  -  Based  upon  the  developing  database 
of  SDS  software  data,  the  preliminary  predictions  and  assessments  can  be  validated.  This  activity 
of  preliminary  validations  may  have  to  continue  using  borrowed  data  until  enough  SDS  data  are 
collected  but  the  "learn  as  you  go"  aspect  of  software  metrics  application  must  be  duly  considered. 

Pilot  Requirements  Model  -  As  a  necessary  aid  to  provide  a  timely  foundation  for 
structuring  early  measurands  from  the  SDS  life  cycle,  a  requirements  model  has  been  proposed. 
Figure  3.1-3  presents  this  model. 

This  proposed  requirements  model  is  identified  that  can  also  serve  as  the  front-end  to 
whatever  life  cycle  model  is  subsequently  developed  consistent  with  Mil-Std-2167A.  The 
requirements  model  can  serve  to  establish  formal  relationships  between  measurands  and  be  used  to 
establish  formal  metrics  in  assessing  development  risk,  and  predicting  cost/schedule  effectiveness. 
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Figure  3.1-3  Pilot  Requirements  Model 


3.1.3  Long-Range  Development 


As  the  experience  and  database  with  the  available  models  and  metrics  matures,  the 
importance  of  advanced  models  with  actual  SDS  factors  and  criteria  grows.  The  activities  planned 
for  long-range  development  are: 

a.  Full  SDS  Requirements  Modeling 

b .  Advanced  Reliability  Modeling 

c.  Survivability,  Integrity,  Portability,  Usability  and  Reusability  Modeling 

d.  Composite  Metrics  Modeling 


These  activities  are  discussed  in  the  following  paragraphs. 
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Full  SDS  Requirements  Modeling  —  This  activity  will  start  with  the  pilot 
requirements  model  and  use  a  formal  method  to  represent  the  SDS.  This  model  should  be  readily 
extensible  into  the  software  architecture  and  detail  design.  The  model  should  include  the  attributes 
presented  in  Table  3.1-2. 

The  pilot  requirements  model  proposed  (Figure  3.1-3)  has  a  robust  set  of  requirements 
generators  and  the  ability  to  serve  as  the  front-end  to  diverse  types  of  life  cycle  models  (iterative 
and  non-iterative).  The  model  allows  requirements  parsing  to  enable  the  proper  allocation  and 
instrumentation  of  requirements  for  subsequent  metrics  measurements  and  assessments.  Several 
dozen  newly  identified  measures  and  measurands  (Table  3.1-1)  have  been  identified.  These  are 
categorized  as  predictive  measurands  and  are  intended  to  provide  early  visibility  into  requirements 
synthesis,  maturity  and  stability.  The  latter  two  are  identified  as  predictive  indicators  that  can  serve 
to  establish  acceptable  thresholds  capable  of  supporting  design.  The  identification  of  too  many 
unstable  and  immature  requirements  preclude  design  activities,  increases  development  risk  and 
creates  an  incomplete  design  envelope). 

To  support  the  model,  it  is  also  proposed  that  formal  mathematical  relationships  between 
model  components  be  identified.  An  initial  attempt  at  formal  mathematical  relationship  has  been 
accomplished  in  conjunction  with  George  Mason  University. 
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Table  3.1-2  SDS  Requirements  Model  Attributes 

Multi-level  Requirements  Identification  —  Computer-assisted  requirements  identification  in  a 
separable  document  basis. 

Iterative  Requirements  Accounting  —  Computer-generated  matching  of  low-to-high  with  high-to- 
low  trace  completion  reporting. 

Incremental  Baseline  Reporting  —  Computer-assisted  baseline  identification  with  computer¬ 
generated  baseline  accounting. 

Full  Real-Time.  Data  Driven  Requirements  —  Computer-assisted  definition  of  all  required  timing 
constants,  data  paths  and  anticipated  control  paths  with  computer  generated  diagnostics  and  data 
dictionary. 

Hierarchical  Decomposition  of  System  Elements  —  Computer-assisted  hierarchy  definition; 
computer  generated  hierarchy  reporting. 

User-Efficient.  Secure  Storage  and  Retrieval  —  Easy  to  use  menus,  helpful  messages,  meaningful 
diagnostics  with  fully  supported  configuration  management. 

Functional  Simulation/Modeling  —  Through  computer  workstation  generated  database  of  the 
system  functions  -  computer-assisted  system  modeling  and  simulation  studies. 

Document  Generation  --  Computer  workstation  generation  of  system  functional  documents. 

Ada  Language  Orientation  -  Computer  generated  prototype  code  or  computer-assisted  integration 
of  design  and  code  elements  with  requirements  and  design  specifications. 

Life  Cycle  Meta-Model  -  A  life  cycle  meta-model  (Figure  3.1-4)  is  required  to 
support  the  coupling  of  the  requirements  model  with  prototyping  and  reuse  life  cycles,  as  well  as 
technology  or  automation-based  paradigms.  The  life  cycle  meta-model  is  proposed  to  illustrate  a 
feasible  approach  to  support  the  various  types  of  development  approaches  (e.g.,  evolutionary, 
incremental,  software  first).  The  meta-mode!  is  intended  to  provide  the  framework  for  the 
measurement  of  productivity,  the  identification  of  associated  products  relative  to  the  measurement 
of  processes,  and  the  elaboration  of  new  design  review  form  factors. 
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Figure  3.1-4  Life  Cycle  Meta-Model 


The  life  cycle  building  blocks  of  Figure  3.1-4  are  used  to  map  phases  of  one  life  cycle 
form  into  another,  similar  to  that  accomplished  in  TR-9033-2  for  different  waterfall  models  and 
variations.  The  model  is  also  required  to  assist  in  the  association  and  parsing  of  categorization  data 
of  Table  1.6-6  of  report  TR-9033-2  across  the  resulting  new  life  cycle  phases. 


Advanced  Reliability  Modeling  -  TR-9033-2,  Paragraph  4.2  described  potential 
shortfalls  in  available  reliability  models  for  the  SDS.  In  particular,  the  lack  of  criticality  factors  and 
the  treatment  of  all  defects  as  unpredictable  were  discussed.  On  a  positive  note,  Paragraph  1.1.6 
of  that  report  relayed  recent  extensions  to  domain-based  model  to  include  the  notion  of  error 
tolerances  in  user  service.  Further  development  to  include  scored  tolerance  would  also  include 
the  concept  of  criticality  (lack  of  tolerance). 


In  a  recent  article,  Jeanette  Wing  and  Mark  Nixon  have  forecast  that  further  extensions  to 
the  "Ina  Jo"  formal  specification  language  may  include  properties  such  as  "fault  tolerance, 
reliability,  performance  and  real-time  behavior."  These  extensions  may  be  applicable  for  the  SDS 
reliability,  survivability  and  integrity  modeling. 
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Another  valuable  extension  for  SDS  is  to  add  software  metrics  data  instrumentation  to  a 
formal  design  language  (e.g.,  "IORL”  from  Teledyne  Brown  Engineering),  and  then  use  SD1 
project  data  (such  as  SIE  and  N-SITE)  to  make  experimental  coupling  and  complexity 
assessments,  and  selected  predictions.  The  experimental  data  can  be  used  to  generate  preliminary 
norms  for  SDS  predictions  and  assessments. 

Survivability,  Integrity,  Portability,  Usability,  Reusability  Modeling  - 
These  factors  require  fresh  development.  Each  of  these  areas  was  determined  to  be  poorly 
supported  by  metrics  without  formal  models,  (as  represented  by  TR-9033-2  Appendix  B  charts  01- 
36). 


Research  into  these  models  should  be  promoted  as  feasible  with  priority  upon 
Survivability,  Integrity  and  Reusability. 

Composite  Metrics  -  The  development  of  multivariate  vector  metrics  has  been  applied 
to  the  field  of  general  systems  optimization  modeling. 

3.2  VALIDATION  OF  ADDITIONAL  MODELS  AND  METRICS 


The  development  of  new  and  refined  models  sets  the  basis  for  requirements  for  a 
validation  testbed  or  suite  consisting  of:  a)  validation  methodology  base,  b)  SDS  software  data  and 
selected  test  data,  and  c)  a  validation  tool  set.  The  validation  base  should  provide  refinements 
guidance  to  further  development. 

3.2.1  Validation  Methodology 


The  first  part  of  the  validation  suite  is  the  methodology  base.  These  are  at  least  two 
exemplary  studies  providing  guidance  for  software  reliability  validations  Abdel-Ghaly,  Chan  and 
Litdewood  compared  the  predictive  quality  of  ten  reliability  models  using  multiple  criteria  (u-plot, 
y-plot,  scatter  plots,  noise  measures,  and  Prequential  likelihood).  John  Bowen  ran  goodness-of- 
fit  "GOF"  convergence  and  reasonableness  tests  upon  the  eight  reliability  models  in  the  SMERFS 
package.  These  statistical  examples  should  serve  as  starting  methods  for  the  validation 
methodology. 
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Prior  studies  of  complexity  models  also  must  be  considered  with  the  SDS  metrics 
validation  methodology.  Virginia  Gibson  and  James  Senn  used  simple  comparisons  to  rank  six 
different  complexity  models.  John  Stephen  Davis  and  Richard  J.  LeBlanc  conducted  tests  of  nine 
different  complexity  measures  using  three  different  programming  environment  databases.  Dermis 
Kafura  and  Geereddy  R.  Reddy  used  simple  comparisons  to  study  seven  different  complexity 
models. 


By  establishing  an  eclectic,  extensible  methodology,  the  cost  of  methodology 
development  can  be  kept  low  and  yet  the  responsiveness  to  the  validation  task  can  be  ensured. 

3.2.2  SDS  Software  Data  and  Selected  Test  Data 

Initially  the  data  io  use  for  validations  may  be  obtained  from  SDI  rapid  deployment 
experiments  (e.g.,  SIE  and  N-SITE).  Additional  project  data  may  be  borrowed  from  the  Rome  Air 
Development  Center  (RADC)  the  Joint  Surveillance  System  (JSS),  the  Naval  Underwater  Systems 
Command  (NUSC)  or  other  defense  systems  agencies  with  similar  programs. 

Later,  as  the  SDS  preliminary  application  of  metrics  proceeds,  the  database  will  contain 
prior  SDS  software  metrics  life  cycle  phase  data  (e.g.,  error  rates,  productivity  levels). 

3.2.3  Validation  Tool  Set 

Two  widely  used  statistics  packages  are  SPSS  and  SAS.  Either  offers  a  PC-based 
version  with  interfaces  to  mainframe  services.  Alternatively,  a  basic  spreadsheet  package  like 
"Lotus  1-2-3"  or  "Excel"  provides  a  simple  tool  set  to  make  iterative  comparisons  of  test  data  and 
statistical  derivations. 

Either  the  statistical  packages  or  the  spreadsheet  packages  have  graphics  capabilities.  The 
former  packages  produce  selected  statistical  functions  much  more  effectively,  but  the  spreadsheet 
packages  can  make  most  types  of  popular  graphs  (pie,  bar,  line,  etc.). 
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3.3  COMPREHENSIVE  MEASUREMENT  METHODOLOGY  DEVELOPMENT 

AND  INTEGRATION 

Section  3.0  of  TR-9033-1  documents  the  methodology  requirements  for  software 
metrics.  The  initial  methodology  requirements  presented  there  emphasize  the  need  for  early 
introduction  of  sofitwaie  measurement  in  the  software  development  life  cycle.  Just  as  important, 
the  software  measurement  process  should  not  be  "tacked  on"  to  the  development  methodology,  but 
integrated  with  it.  Inherent  in  the  integration  of  the  software  measurement  methodology  with  the 
software  development  methodology  are  well-developed  review  points.  With  the  information 
presented  at  these  review  points,  not  only  can  redirection  of  design  be  undertaken,  but  also 
redirection  to  meet  quality  goals. 

With  the  above  points  in  mind  the  following  tasks  need  to  be  completed: 

a)  Definition  of  a  standard  prototyping  methodology  for  SDS  software 
development. 

b)  Integration  of  the  standard  prototyping  methodology  with  the  Software 
Measurement  Program. 

c)  Development  of  a  standard  methodology  for  rating  quality. 

d)  Development  of  a  standard  software  metrics  model  "tuning"  methodology. 

e)  Development  of  a  standard  metrics  selection  methodology. 

It  is  envisioned  that  prototyping  activities  (including  Rapid  Prototyping)  will  be  a  vitally 
important  component  of  SDS  development.  Today,  the  prototyping  approach  is  not  well  supported 
by  standards  and  guidelines,  as  is  the  classic  water  fall  development  process.  Standards  and 
guidelines  are  necessary  to  specify  the  objectives  of  each  development  phase,  as  well  as  the 
corresponding  audit  and  review  activities.  As  noted  in  TR-9033-1,  prototyping  comes  in  at  least 
two  types:  Throw-away,  and  Evolutionary.  Both  types,  and  intermediate  variations,  must  be 
accommodated.  Specifically  to  be  addressed  is  the  consideration  that  contractors  may  already  have 
differing  prototype  approaches  in  place,  and  may  be  reluctant  to  change  them. 

As  part  of  defining  the  prototyping  methodology  for  SDS,  software  metrics  should  be 
included.  The  integration  would  provide  a  methodology  analogous  to  that  for  the  waterfall  model, 
with  software  metrics  auditing  and  review  activities  occurring  concurrently  with  standard  audit  and 
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review  activities.  One  of  the  intriguing  software  metrics  proposed  is  a  metric  that  predicts  when 
prototyping  will  be  complete,  and  further  iterations  or  modifications  to  the  prototype  are  no  longer 
necessary  or  cost-effective. 

Current  approaches  to  rating  the  quality  attributes  of  a  software  product,  including  those 
for  the  rating  criteria,  are  poorly  defined.  To  achieve  uniform  ratings  among  different  users,  and  to 
obtain  ratings  which  are  reliable  measures  of  the  extent  to  which  a  software  product  meets  its 
quality  requirements,  a  well-defined  rating  methodology  is  necessary.  It  is  also  required  for  tuning 
the  quality  metrics,  so  that  their  predictive  capabilities  can  be  made  as  accurate  as  possible. 


The  validation  of  software  metrics  require  that  they  produce  outputs,  or  scores,  which 
can  predict  the  quality  rating  of  the  final  product  with  acceptable  accuracy  and  consistency. 
Validation  of  a  software  metric  is  complex,  and  requires  tuning  the  model  to  improve  its  predictive 
accuracy  until  it  is  acceptable.  This  tuning  process  must  be  repeated  across  each  individual 
software  domain.  Because  tuning  could  require  modification  of  literally  hundreds  of  different 
elements,  metrics,  criteria,  rankings,  etc.,  possibly  in  combination,  it  cannot  be  accomplished  by 
any  straightforward  method.  It  is  therefore  necessary  to  establish  a  methodology  for  software 
metrics  tuning  so  that  convergence  to  a  stable  configuration  of  the  metrics  can  '  ,■  achieved  as 
expeditiously  as  possible.  The  methodology  may  be  both  analytical  and  experimental.  Analytical 
methods  may  be  used  for  identifying  strategies,  based  on  the  analysis  of  the  correlations  among  the 
quality  factor  scores,  the  quality  ratings,  the  applicable  metrics,  and  metric  elements.  Experimental 
methods  would  be  used  to  analyze  the  effects  of  the  tuning  strategies  on  a  variety  of  software 
programs  within  the  same  software  domain.  Selection  of  the  tuning  strategies  may  be  based  on: 

a)  analytical  methods; 

b)  random  searches  in  the  metrics  space; 

c)  past  experience. 


When  a  development  effort  is  begun,  the  selection  of  the  appropriate  quality  goals, 
factors,  rankings,  and  tools  could  be  a  daunting  prospect.  Consequently,  the  guidelines  and 
recommendations  for  approaches  documented  in  TR-9033-1  need  to  be  formalized  into  a  system 
that  will  aid  in  the  selection  of  the  proper  software  measurement  parameters.  While  the  system 


should  eventually  be  automated,  even  a  manual  implementation  of  a  selection  methodology  would 
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be  a  significant  step  forward.  The  methodology  would  guide  the  user  in  first  setting  the 
appropriate  quality  goals  for  the  development.  It  would  assist  in  determining  the  software 
domain(s)  in  wmch  the  development  best  fits.  Through  analysis  of  criticality  level,  and  tradeoff  of 
related  software  metrics,  the  system  would  help  the  developer  in  defining  a  baseline  set  of  software 
metrics  for  his  use.  While  such  a  system  would  always  require  final  determination  of  the 
selections  based  on  overall  considerations,  such  a  formalized  approach  would  remove  much  of  the 
guesswork  from  the  software  metrics  selection. 

3.4  DEFINITION  AND  DEVELOPMENT  OF  AUTOMATED  TOOLS 

AND  DATABASES 

The  process  of  tailoring  the  selected  tool  packages  (SMERFS,  SMART,  SECOMO 
and/or  REVIC)  into  a  smoothly  operated,  user-friendly  management  environment  is  described  in 
this  section. 

Figure  3.4-1  presents  an  overview  of  the  activities  to  develop  (or  later,  enhance)  the  tools 
environment. 

3.4.1  Tool  Acquisitions  and  Installations 

Each  of  the  tools  must  be  properly  acquired  and  installed.  Specific  setup  procedures 
must  be  checked.  Options  must  analyzed.  Alternative  installation  sequences  should  be  tested. 
This  step  can  be  ad  hoc  based  upon  tool  and  resource  availabilities. 

A  tentative  order  to  bring  in  tools  is: 

a.  SMERFS 

b.  SECOMO  or  REVIC 

c.  SMART 

d .  AdaMat  or  AdaCAT 

e.  Others 
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SMERFS 


Figure  3.4-1  Software  Measurement/Environment  Tool  Development  Activities 
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3.4.2  Too!  Prototype  Use  and  Evaluation 

As  each  tool  is  successfully  installed,  it  can  then  be  scheduled  for  prototype  use,  data 
acquisition  and  evaluation  with  sample  SDS  data.  For  some  tools,  e  g.,  SMERFS,  it  may  be 
necessary  to  borrow  or  fabricate  test  data.  For  others  e  g.,  SECOMO/REVIC  live  data  may  be 
applicable. 

A  preliminary  use  and  evaluation  method  will  be  drafted  for  SMERFS,  SECOMO/REVIC 
and  SMART.  Later  this  method  will  be  reviewed  for  expansion  or  revisions.  Some  preliminary 
evaluation  criteria  include: 

a.  Tool  input  requirements  availability 

b .  User  friendliness 

c .  Report  readability 

d .  Processing  cycle  responsiveness 

e .  Adequacy  of  data  integrity  controls 

Based  upon  the  specific  evaluations  for  each  tool,  selected  utility  command  files  and/or 
utility  programs  may  be  indicated.  For  example,  the  SMERFS  package  provides  subroutine 
interface  descriptions  to  allow  a  tailored  user-specific  executive.  Alternatively,  the  off-the-shelf. 
Executive  may  only  require  its  incorporation  within  a  menu-based  software  selection  package  like 
MicroSoft  Windows  or  a  customized  menu  written  in  Clipper,  dBase  Pascal  or  C. 


3.4.3  Tool  Interface  and  Database  Analysis 


For  those  tools  requiring  selected  enhancements,  one  or  more  of  the  following  may  be 

required: 

a)  Command  &  Parameter  Analysis  --  a  very  simplistic  analysis  for  program 
package  which  can  be  driven  from  a  menu  handler. 

b)  Program  Interface  Analysis  —  a  more  intense  analysis  to  build  or  adapt  an 
executive  module  or  a  low-level  hardware  driver  to  new  user  or  hardware 
demands. 
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c)  Database/File  Format  Analysis  --  a  moderate  analysis  to  develop  or 
enhance  data  extraction  services  downstream  of  a  package  (e.g.,  SMART 
to  Lotus  or  to  Excel). 

3.4.4  Environment,  Procedures  and  Shell  Maintenance  or  Specification 

This  activity  initially  may  be  a  modest  programming  activity  to  install  a  menus  package. 
Later,  when  it  becomes  necessary  to  add  tools  or  to  adjust  the  tool  selections,  the  changes  should 
be  very  brief  to  make  and  document. 

3.4.5  Procedures  Development 

As  each  tool  set  is  integrated  into  the  software  metrics  environment,  the  set  of  user 
procedures  should  be  drafted.  As  each  tool  becomes  used,  the  procedures  should  be  reviewed  and 
revised  if  needed. 

3.4.6  Shell  Command  Files  Development  and  Test 

The  computer  commands  to  provide  user  efficiency  and  smooth  integration  must  be  built 
and  tested  by  technical  support  staffs.  The  data  used  to  test  each  tool  should  be  preserved  for  user 
tutorial  sessions. 

3.4.7  Utility  Development  and  Test 

As  appropriate,  utility  programs  should  be  coded  in  a  popular  language  like  Turbo  Pascal 
or  Turbo  C.  Simple  testing  should  precede  complex  tests. 

3.4.8  User  Environment  Testing  and  Acceptance 

After  each  of  the  models  has  been  unit-tested  with  its  applicable  adaption  data,  the  total 
environment  must  be  addressed.  The  users  will  be  asked  to  pass  judgement  on  the  smoothness  of 
program  initiation  and  execution. 
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